Speech, Space and Purpose: Situated Language Understanding in Computer Games

Many common types of language understanding depend on situational context. In the extreme, utterances like ”the one to the left of the green ones”, ”let’s do that again” or ”can you help me?” provide little content or restrictions through their words, but can be readily understood and acted upon by a human listener embedded in the same situation as the speaker. We describe a series of computational models of situated language understanding that take into account the context provided by a game the language users are playing. Starting with a game focusing on spatial disambiguation, we proceed to a model taking into account player’s recognized intentions to perform referent disambiguation and end with a system that understands highly situated commands directly in terms of recognized plan fragments. Finally, we discuss our use of these models in building artificial agents that plan alongside the player in the game world and co-operate through language and their own initiative.